HDDS-14183. Attempted to decrement available space to a negative value#9655
HDDS-14183. Attempted to decrement available space to a negative value#9655sumitagrawl merged 8 commits intoapache:masterfrom
Conversation
|
This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days. |
siddhantsangwan
left a comment
There was a problem hiding this comment.
It seems to me that the root cause is not bad exception handling, but actually not implementing idempotency correctly in the overwrite case.
In FilePerBlockStrategy#writeChunk, containerData.updateWriteStats(chunkLength, overwrite) is called at the very end. At that point any exception is already thrown. So it's correct to update write stats at that point.
And updateWriteStats always calls incrWriteBytes(bytesWritten), which always calls volume incrementUsedSpace(bytes):
public void updateWriteStats(long bytesWritten, boolean overwrite) {
getStatistics().updateWrite(bytesWritten, overwrite);
incrWriteBytes(bytesWritten); <--------------- this, always called
}
private void incrWriteBytes(long bytes) {
this.getVolume().incrementUsedSpace(bytes); <-------------- this, always called
long bytesUsedBeforeWrite = getBytesUsed() - bytes;
long availableSpaceBeforeWrite = getMaxSize() - bytesUsedBeforeWrite;
if (committedSpace && availableSpaceBeforeWrite > 0) {
long decrement = Math.min(bytes, availableSpaceBeforeWrite);
this.getVolume().incCommittedBytes(-decrement);
}
}
So used space is increased and available space is decreased even on overwrite. For overwrites that don't actually grow the file on disk, this is wrong and it's probably what leads to available space being decreased more than what's possible.
For example:
chunkLength = 1,048,576 (1MB),
block file already length 1,048,576,
retry writes same chunk at offset = 0 (overwrite = true),
so after write, file length still 1,048,576 (no growth). But used space is increased and available space is decreased incorrectly.
So this is what needs to be fixed. I also suggest trying to keep it simple as we are already using the term reserved space in other parts of the code, and that means something else. This area is pretty complex at this point!
|
This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days. |
siddhantsangwan
left a comment
There was a problem hiding this comment.
LGTM. @sarvekshayr If you can add some code comments to explain how this part works, I think that'd help other reviewers reason about it as well.
|
Thanks @sumitagrawl, @siddhantsangwan, @yandrey321 and @sreejasahithi for the review. |
What changes were proposed in this pull request?
Saw this warning when datanode disk was nearly full:
Prior to this message, there were many failed writes. Perhaps it needs to increment the value when the write fails.
The fix adds rollback logic in
KeyValueHandler.handleWriteChunk()that tracks when a chunk write succeeds and increments theusedSpacecounter. If any subsequent operation fails, the exception handler callsvolume.decrementUsedSpace()to restore the counter.What is the link to the Apache JIRA
HDDS-14183
How was this patch tested?
CI: https://github.com/sarvekshayr/ozone/actions/runs/21200210393